Loading...
 

MorphIT fault diagnosis

Error diagnosis under MorphIT

This article is a guide to detecting and fixing errors in MorphIT.

Page not responding / server not responding

Possible causes:

Server does not come out of maintenance mode

Possible causes:

solutions:

If the server is in test mode, then the Admin Console should have the entry test mode: enabled in the status command in the section == Server ==. In this case, the test mode can be deactivated again simply by using the command test_mode disable via the Admin Console.

Server does not start

Possible causes:

  • NodeJS dependencies are not up-to-date
  • Syntax error in config.js
  • One of the used ports is already from another program
  • The server was started multiple times or a previous server process did not terminate correctly
  • The service has not yet been started after a restart because it starts with a delay
  • The service is set up incorrectly because paths have changed since installation

solutions:

If the server cannot be started because it stops immediately, the cause of the error can be found in the server log (console output). If the server starts as a service, the server log is normally written to .../WebService/morphIT/logs/server.log .

The file .../WebService/morphIT/custom/config.js must be a valid JavaScript file. A syntax error in the file causes the server not to start. In this case the log contains a corresponding error message.

To update the NodeJS dependencies, only the script .../WebService/morphIT/setup_dependencies.bat must be executed. (Internet connection required)

If a used port is already in use, this should be noted in the server log. The only thing that helps here is to set up other ports in .../WebService/morphIT/custom/config.js or to terminate the application that is causing the problem.

If a leading server process has not finished completely, it can be terminated manually via the Task Manager. Process name: node.exe

If the server is configured as a service (Name: ClassiX Web-Server, ID: cxserver), this service may not have been started yet. After a restart this is normal, because the service starts delayed. The service can be started manually in the service administration(services.msc). If this fails, although the server can be started manually (.../WebService/morphIT/start_server.bat), the service is possibly configured incorrectly. In the event display of Windows you should then find error messages from nssm (our service process) describing why the service could not be started. The easiest way to solve the problem is to remove and reinstall the service. For this purpose the scripts .../WebService/morphIT/uninstall.bat & .../WebService/morphIT/install_service.bat must be started with administrator rights.

Client does not get out of the queue

The server is started and the client can connect to the server and is queued for a ClassiX instance, but after that nothing happens.

Possible causes:

solutions:

The easiest way to determine the cause of this problem is to use the Admin Console. The status command provides information about the current status of the server.

In the section === Launch Strategy === you should look for a red entry. If the status is disabled, the server is not configured to restart ClassiX instances on its own. In the .../Webservice/morphIT/custom/config.js launch_strategy.enabled must be set to true .
If the entry reserve target is set to 0 , the configuration value launch_strategy.min_free_instances must be set to at least 1 , otherwise no instances will be kept.
If the entry depleted is found under licenses, the ClassiX licenses are used up and therefore no new instances can be started.
If the entry under launchers is depleted, then no launcher is connected, or the connected launchers are already at the limit of the instances to be started. If no launcher is connected, then the section === Launcher ===the entry is connected: 0. In this case it is possible that

If a launcher is connected and the launcher capacity is exhausted, the section == Launcher === can be used to control for each launcher how many instances it can start and how many have already been started. The maximum number of instances to be started can be set for each launcher in the file .../Webservice/launcher/custom/launcher.js using the key max_instances (default: 4). After the configuration has been adjusted, it is sufficient to restart the launcher service (name: ClassiX Launcher, ID: cxlauncher). The server does not have to be restarted for this purpose. Instances that have already been started are not terminated by this. The server is not affected by a launcher restart.

Client cannot bind himself to interactive ClassiX

The server is started, an interactively started ClassiX instance is connected to the server (verified by admin console), but MorphIT cannot connect to the instance.

causes:

  • Connection to interactive ClasisX not configured

solutions:

In the .../Webservice/morphIT/config.js the key ws.classix.interactive.connect_to_morphit is false by default, because this is mainly needed for the development of MorphIT and should be deactivated in productive operation. In the .../Webservice/morphIT/custom/config.js this value must be overwritten with true, so that the server allows a connection between interactive ClassiX and MorphIT client. This option has no influence on the use of native WebWidgets.

Client does not display font icons or does not display them correctly

Icons Werden Nicht Dargestellt

Possible cause:

  • The font download is deactivated in the browser security settings.

solution:

  • Internet Explorer:
    Internet options -> Security -> Internet -> Adjust level
    In the settings, search for downloads and activate font download.
    Then reload the page.

Kontext IE Internetoptionen Sicherheit IE Stufe Anpassen Einstellungen IE

Client does not display images

No images are displayed in the client, or some images are not displayed.

causes:

  • The picture simply does not exist
  • The asset configuration of the server is not complete.
  • Launcher for remote assets not started
  • The configuration of the Remoate Asset Launcher does not match the server configuration

solutions:

If the image simply does not exist, it will not be displayed even if the application is opened natively.

If the image is displayed natively, it may be because the asset configuration is not complete. To verify that this is causing the problem, open the developer tab in MorphIT (F12) and then reopen the page where the image is missing with the network tab open. If a request with the path .../missing_path_map_entry appears in the logged HTTP requests, then the ClassiX process complains that the MorphIT server has not defined a path mapping for the requested image path.
In order to translate a path such as Y:\ClassiX\Evaluate\icons\test.png into /public/assets/images/test.png, ClassiX needs information from the MorphIT server about which local paths are to be translated into which web paths, and this information must be provided here for the requested path.

The searched path is written by ClassiX as a warning in the log in the form
No matching entry in path map found for asset path: "c:\classix\cyberenterprise_000005\icons\cx_keyword.png

If the image URL requested by the browser looks reasonable but no image is loaded, and the server has been configured for remote assets, then the error may simply be that the launcher responsible for it has not been started, or the host in the path configuration contains a typo. In this case, the server responds with an HTTP 503 response. And if you call up the URL manually in the browser, or look at the response in the network tab, you will see the following response generated by the server, which also outputs for which host the launcher is missing:

No launcher available for host #host# to serve this request.

If the launcher has been started, check if it has connected to the server(status command in the admin console) and if not, proceed as described here.

If the MorphIT server uses remote assets and instead of the 503 error message from the server the image is simply not loaded (the network tab then says Status: Request pending), then the launcher is connected but cannot transfer the file to the server. This is mainly caused by the fact that port in the config.js of the launcher does not match the port of the config.js of the server, because the request to send a file is made by the server via the websocket connection, which runs via the ws.launcher.port, but the file itself is transferred via an HTTP put request from the launcher to the server and this is started on the port configured under port. If this is the problem, then the launcher will display error messages in the console as soon as the images are requested.

Client does not show error details in case of error message

If an error occurs in the InstantView code, the error message is shown in the MorphIT client, but no code line is displayed and no call stack. The "Yes" buttons (for further details) are also not displayed.

Causes:

Solution:

This is intentional. For security reasons, ClassiX instances started by the launcher should not give out internal information (such as a call stack) in an error message to a MorphIT client. This mechanism does not apply to interactively started ClassiX instances.

For troubleshooting, either the log should be evaluated or the error should be reproduced with an interactively started instance.

Native WebWidget does not show

When you open the WebWidget, the MorphIT background appears, but not the WebWidget

causes:

  • WebWidget port is blocked
  • Error in WebWidget-Code
  • Error in MorphIT code
  • ClassiX does not answer

solutions:

For all possible causes of errors a look into the developer console of the browser (F12) helps. In Internet Explorer, the page must be reloaded after opening the console so that any error messages are displayed. If the console contains an error message, then it should be possible to deduce from the message which of the first three points applies.

If there is no error message in the console, it is possible that ClassiX does not respond to the messages of the web widget (possibly not implemented). For this you have to examine the individual message frames of the opened web socket and compare them with the protocol of the web widget . Chrome offers this possibility directly, for Firefox there is the extension: WebSocket Monitor.

Launcher does not start

Possible causes:

  • NodeJS dependencies are not up-to-date
  • Syntax error in configuration files
  • Start command not defined
  • The service is set up incorrectly because paths have changed since installation

solutions:

If the launcher cannot be started because it stops immediately, the cause of the error can be found in the launcher log (console output). If the launcher starts as a service, the launcher log is normally written to .../Webservice/launcher/logs/launcher.log

To update the NodeJS dependencies, only the script .../Webservice/launcher/setup_dependencies.bat must be executed. (Internet connection required)

The configuration files .../Webservice/launcher/config/custom/launch.js & .../Webservice/launcher/config/custom/launcher.js must be valid JavaScript files. A syntax error causes the launcher to fail to start. Such an error will be noted in the log accordingly.

In the configuration file .../Webservice/launcher/config/custom/launch.js a field cmd must be defined, which specifies the start command (e.g. path to batch file), with which an instance can be started. If this entry is missing, the launcher will not start either. The error is noted in the log.

If the launcher is configured as a service (Name: ClassiX Launcher, ID: cxlauncher), this service may not have been started yet. After a restart this is normal, because the service starts delayed. The service can be started manually in theservices administration(services.msc). If this fails, although the launcher can be started manually (.../Webservice/launcher/start_launcher.bat), the service is possibly configured incorrectly. In the Windows Event Viewer you should then find error messages from nssm (our service process) describing why the service could not start. The easiest way to solve the problem is to remove and reinstall the service. For this purpose the scripts .../Webservice/launcher/uninstall.bat & .../Webservice/launcher/install_service.bat must be started with administrator rights.

Launcher cannot connect to server

The launcher can be started, but in its log (console output) you can only see that the connection to the server fails

Possible causes:

  • Server is not started
  • Connection configuration from launcher does not match the server
  • Firewall blocks the connection

solutions:

At startup, the server outputs the connection information in the server log (console or under .../WebService/morphIT/logs/server.log). There you will find the following line: Launchers should connect to ws://localhost:8081/morphit/launcher

If the server configuration has not been adjusted in this respect, then the launcher is correctly configured from the outset (if server and launcher are running on the same machine). The configuration of the Launcher is composed of the .../Webservice/launcher/config/launcher.js and their values can be overwritten in the .../Webservice/launcher/config/custom/launcher.js. The correct configuration can be derived from the above connection string as follows:

If the string starts with wss://, then ssl:true otherwise(ws://) ssl:false.
host must be set to the host name of the computer on which the server is running (default: "localhost")
port is the number that follows the host name in the connection string (after the colon) (here: 8081)
endpoint is the rest of the connection string (Here:"/morphit/launcher")

If launcher and server are running on different machines, the firewall can additionally prevent the connection from being established.

Launcher cannot start ClassiX instances

The launcher is connected to the server, but the start of an instance fails. In the server log there is no connection attempt of a ClassiX instance.

causes:

solutions:

First of all, it must be ensured that the ClassiX instance can be started regularly (via the CMD) using the launcher command(cmd from .../WebService/launcher/config/custom/launch.js).

A file redirection can also be specified in cmd. For example, start_classix.bat > launch.log can be used to redirect the console output of the start batch to a file for debugging. This is useful for troubleshooting problems that only occur when starting via the launcher. Access to network drives and user resources can cause problems, because the launcher normally runs as a service process and under another user with different rights and probably without mounted network drives. The launched sub-processes run under the same user and are therefore subject to the same restrictions.

If it is possible to start ClassiX, but you can close it directly, the corresponding ClassiX logfile should be searched for the corresponding error.

Server rejects newly started ClassiX instances

The launcher is connected to the server and the server starts new instances via the launcher, but these are rejected by the server as soon as they connect to the server.

causes:

solution:
If it is not due to incompatible dlls and OpenMorphITConnection was not called manually with the wrong parameters or the environment variables used for this purpose(CX_MORPHIT_... ) were overwritten, then the connection should be established without any problems.

Error message: Launched process terminated after only ... ms

After the server starts a new instance via a launcher, this error message appears in the server log and launcher log, although the ClassiX instance was started correctly and a connection is possible.

causes:

  • Incorrectly configured batch file

solutions:

As described in the error message, this message is caused by an incorrectly configured start batch file. This causes the connection between the ClassiX instance to be lost, which means that the launcher does not know how many instances have really been started(max_instances is bypassed) and furthermore the launcher cannot manually terminate a non reacting ClassiX instance. The problem can also be diagnosed via the admin console, because there you can see in the status that ClassiX instances are started and in the ClassiX instance you can see from which launcher it was started, but the launcher itself says that it didn't start any instances.

The launcher checks at regular intervals which of its started processes are still running and thus knows how many instances can still be started. If in the batch file to be started a ClassiX process is started with START without specifying the parameter /WAIT, the process started by the Launcher ends immediately and the Launcher can no longer terminate the ClassiX process (if it hangs) via the process ID. This may lead to the fact that the server cannot be successfully put into maintenance mode.

Since it can quickly happen that the start batch is incorrectly configured, the server must consider this possibility and accept the connection from the incorrectly started instance anyway. It can only do this by ignoring the signal from the launcher (that the instance has finished) and waiting for a timeout(ws.launcher.launch_timeout) before it can assume that the instance was not started successfully. This timeout makes the launch process more inefficient than it could be.

Therefore the start scripts should be written in a way that they do not terminate while the ClassiX process is still running. If the open console window is undesirable during normal operation, the environment variable CX_START_WAIT can be used for this purpose. The Launcher sets this variable to"/WAIT " for its subprocesses.

Admin Console does not start

causes:

solutions:

If the start_console.bat reports: "Please run the batch file setup_dependencies.bat before installing the service", then the NodeJS dependencies are missing. These can be downloaded from the Internet via the script setup_dependencies.bat. For this the computer must also have access to the Internet. If something should fail here, then a npm-debug.log file is stored in the directory from which the error can be seen. If the computer must not have access to the internet, then the script can be executed on another computer and then the generated folder node_modules can be copied to the target computer.

Admin Console cannot connect

causes:

  • There is no MorphIT server running
  • The console was not started on the same machine
  • The Admin Console has been disabled
  • The configured port does not fit
  • The SSL configuration does not fit

solutions:

In the default settings of the MorphIT server, for security reasons only connections from the admin console of the machine on which the server is running are allowed. This setting should not be changed and the console should always run from the same machine.

In the .../Webservice/morphIT/custom/config.js the admin console can be completely deactivated via the entry ws.admin.enabled. This value must be set to true so that the console can establish a connection to the server.

The entry ws.admin.port of the server configuration specifies the port on which the admin console can connect to the server, and this must match the entry endpoint.port from .../Webservice/admin_console/config/config.js, or .../Webservice/admin_console/config/custom/config.js.

If the port configured by the server runs over SSL, then the entry endpoint.ssl must also be set to true in .../Webservice/admin_console/config/custom/config.js.

Maintenance server does not start

The maintenance command has been issued in Admin Console, but it does not report that the Maintenance Server has started successfully.

causes:

  • MorphIT server crashed during shutdown
  • The process for the maintenance server could not be started
  • The maintenance server had an unexpected error when starting

solutions:

Quick solution:

  1. Check in the Windows services and make sure that the MorphIT server (and MorphIT launcher) are closed (if not, close it).
  2. Open CMD and run morphit start mserver (first 4.15.0). The maintenance server is started directly in the CMD and any error messages are directly visible.

Alternative solution:

In the most common cases, it is enough to complete the process:

  1. (re-)starting the MorphIT server
  2. Putting the machine in maintenance mode from the Admin Console

several times to bypass the problem and start the maintenance server. To investigate (and perhaps fix) the problem in more detail, proceed as described in the following paragraphs.

Since the server has a timeout for every single step during shutdown, it is unlikely that the server hangs during shutdown because it is waiting for something. What may have happened is that an unexpected error occurred that caused the server to crash before the maintenance server was started. This error should appear in .../Webservice/morphIT/logs/server.log. Since the server is normally set up as a service and stopping the service is the last step in shutting down, the server should then restart normally. The admin console notices this and automatically resends the maintenance command(up to 3 times). If the server regularly crashes during shutdown, the maintenance server may not be able to be started. The error should be searched for in the server log and corrected.

If the MorphIT server has no rights to start the maintenance server as a new process, you can see this by the fact that a cxt...json file with the server configuration is below %TEMP%. The file is created immediately before the maintenance server is started and is deleted by the maintenance server at startup. For this purpose there should also be an entry in the server log. In this case the maintenance server can be started manually from the CMD: 

node maintenance_server.js --full-config %TEMP%\cxt...json --console

(Parameter "console" only from MorphIT 3.16.5, before MorphIT 4.15.0 the first parameter is simply called "config")
In this way, it is also possible to check via the console output whether the maintenance server itself has errors when starting.

The maintenance server reads the Config-JSON at startup and then, based on the paths/values set there, files are loaded into memory and a web server is started. The maintenance server generates under %TEMP% a log file with the schema cxm...log, which is solved automatically as soon as the maintenance server is replaced by a MorphIT server in a regular way. If the maintenance server has an error at startup, the log file should contain the configuration used for startup and the error that occurred. To restart the maintenance server manually, the configuration (starting from the first opening curly bracket) can be copied into a .json file. The server can then be started manually using the command from the previous paragraph.

ClassiX instances cannot connect to the server

ClassiX instances can be started, but the server log does not show any connection establishment. Not even if the Server and ClassiX are running on the same machine.

cause:

  • Computer has no network and DNS cannot resolve localhost (before Dll version 20013)
  • Firewall blocks access
  • Incorrect connection parameters
  • Call of StartMorphITAutomatically is missing or incorrect

solutions:

If MorphIT is started on a computer without a network connection, it is possible that the connection to localhost will fail because the name resolution of localhost is performed by the now inactive DNS service. The problem can be avoided by using the local loopback address instead of localhost. 20013: This correction is no longer necessary; it is performed automatically by this machine.

Depending on the mode in which MorphIT is used, corrections must be made at a different point.

  • Interactive: In the start batch of ClassiX CX_MORPHIT_HOST=127.0.0.1 must be set
  • Launcher: Under .../WebService/launcher/config/custom/launcher.js set the entry "host": "127.0.0.1".

If the log file of the ClassiX process now shows that the connection is established, but the server rejects the connection, the firewall may be the cause for the problem (even without network). Kaspersky often gets in the way of MorphIT. Disabling the firewall and then restarting the server and ClassiX should solve the problem.

If ClassiX further indicates that it cannot connect, the connection parameters(CX_MORPHIT_HOST, CX_MORPHIT_PORT, CX_MORPHIT_SSL, CX_MORPHIT_ENDPOINT) may not match those of the server. These must be set as the server indicates in its log in the line "ClassiX should connect to: ...".

If ClassiX can be started, but does not establish a MorphIT connection, the statement webService::StartMorphITAutomatically may be missing in the .cxp. This procedure checks if the ClassiX was started by a launcher and reads the parameters from the corresponding environment variables and then establishes the connection to the server.

ClassiX instances report resource problems

During operation, the currently connected ClassiX instance occasionally aborts with the error message that not enough resources are available.

causes:

  • Working memory used up
  • Session heap used up

solutions:

In the Admin Console, execute status--launcher. This will show the amount of memory being used on the machine for each Connected Launcher process. If memory is very limited, the number of ClassiX instances to be started on this machine should be reduced or more memory should be used. The number of instances to be started can be controlled in the .../Webservice/launcher/config/custom/config.js via the entry max_instances

If the working memory in the Admin Console is not fully utilised, then the session heap may simply be used up. Since the launcher normally runs as a service and all services run in session 0, the session heap for this session must be sufficiently large. By default, it is much smaller for Session 0 than for interactive sessions. Before changing anything in the values, it is recommended to read the following post and realise that a wrong configuration can prevent the server from booting.

See also: https://blogs.technet.microsoft.com/askperf/2007/07/24/sessions-desktops-and-windows-stations/ (link is dead)
Updated link: https://techcommunity.microsoft.com/t5/ask-the-performance-team/sessions-desktops-and-windows-stations/ba-p/372473

To increase the heap for session 0, the following entry must be adjusted in the registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems\Windows

This contains a long text which looks like the following:
%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,512Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16

The third number after SharedSection is the size of the session heap for session 0 in KB. To avoid resource problems, this value should be adjusted to the value for interactive sessions (the second number after SharedSection).

This adjustment significantly increases the session heap. In order not to slow down the system, we recommend additionally under the key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
to increase the SessionViewSize entry. The SessionViewSizeentry specifies how much memory (in MB) in total is to be reserved for session heaps. Here we recommend that you set the value of the desktop session heap to four times its original value (here: 4 x 20 = 80).

The computer must be restarted for the changes to take effect.